Firearm simulation and training system and method

ABSTRACT

Disclosed embodiments provide systems and methods for simulation of firearm discharge and training of armed forces and/or law enforcement personnel. A motion tracking system tracks motion of one or more users. In embodiments, the users wear one or more sensors on their bodies to allow tracking by the motion tracking system. A scenario management system utilizes position, orientation, and motion information provided by the motion tracking system to evaluate user performance during a scenario. A weapon simulator includes sensors that indicate position of the weapon and/or orientation of the weapon. The weapon simulator may further provide trigger activation indications to the scenario management system. In embodiments, the scenario management system generates, plays, reviews, and/or evaluates simulations. The evaluation can include scoring based on reaction times, posture, body position, body orientation, and/or other attributes.

CROSS-REFERENCE TO RELATED APPLICATIONS SECTION

This is a continuation Application of application Ser. No. 17/015,425filed on Sep. 9, 2020 which is a continuation Application of applicationSer. No. 15/865,731 filed Jan. 9, 2018. The disclosure of the priorapplication is hereby incorporated by reference herein in its entirety.

FIELD OF THE EMBODIMENTS

Disclosed embodiments relate to firearm simulation and training, andmore particularly to firearm simulation and training utilizing motiontracking.

BACKGROUND

Military training programs of the US Armed Forces are vital for missionsuccess. Military training, aimed at the accomplishment of tasksassociated with a military mission, is an important part of troopreadiness. Without proper training, soldiers may be unprepared for anactual battle. Similarly, law enforcement personnel benefit fromtraining for various situations they may encounter during their patrols.In law enforcement, personnel may not have much advance notice of whenthey will be in a dangerous situation. Thus, periodic training exercisesare important for law enforcement to maintain a state of readiness. Itis therefore desirable to have improvements in the training of militaryand law enforcement personnel.

SUMMARY OF THE EMBODIMENTS

In one embodiment, there is provided a computer-implemented method forconducting a firearm usage simulation, comprising: determining aposition of a user; determining an initial physiological orientation ofthe user; detecting a simulated firearm discharge; and determining areaction of the user in response to the simulated firearm discharge.

In another embodiment, determining a physiological orientation of theuser comprises determining a position of one or more limbs of the user.

In another embodiment, determining a physiological orientation of theuser comprises determining a facial direction of the user.

In another embodiment, determining a physiological orientation of theuser comprises determining an eye gaze direction of the user.

Another embodiment includes comparing a second physiological orientationof the user to an expected physiological orientation of the user; andgenerating a user score based on the comparison.

In another embodiment, comparing the second physiological orientation ofthe user to an expected physiological orientation of the user includescomparing a position of one or more limbs.

In another embodiment, comparing the second physiological orientation ofthe user to an expected physiological orientation of the user includescomparing a facial direction of the user to an expected facialdirection.

Another embodiment includes measuring a duration from a time of thesimulated firearm discharge to a time of detecting the secondphysiological orientation.

In another embodiment, generating the user score is further based on theduration.

In another embodiment, generating the user score is further based on aweapon draw time.

In another embodiment, generating the user score is further based on aweapon discharge time.

Embodiments include a weapon usage evaluation system, comprising: amotion tracking system, configured and disposed to track one or moreusers; a scenario management system, configured and disposed to receivemotion information from the motion tracking system; and a weaponsimulator, wherein the weapon simulator is configured and disposed toprovide discharge information and position information to the scenariomanagement system.

In another embodiment, the system includes an inertial tracking deviceaffixed to the weapon simulator.

In another embodiment, the system includes a shock mount configured anddisposed between the weapon simulator and the inertial tracking device.

In another embodiment, the shock mount comprises: a top plate; a bottomplate; and a plurality of resilient members configured and disposedbetween the top plate and the bottom plate.

In another embodiment, the weapon simulator comprises: a trigger; amotor configured and disposed to generate a vibration in response toactivation of the trigger; a communication module configured anddisposed to wirelessly transmit a trigger indication to the scenariomanagement system in response to activation of the trigger.

In another embodiment, the system includes an inverse kinematic solver.

In another embodiment, the system includes: a processor; a memorycoupled to the processor, wherein the memory contains instructions, thatwhen executed by the processor, cause the processor to perform steps of:determining a position of a user; determining an initial physiologicalorientation of the user; detecting a simulated firearm discharge; anddetermining a reaction of the user in response to the simulated firearmdischarge.

In another embodiment, the memory further contains instructions, thatwhen executed by the processor, cause the processor to perform the stepof determining a position of one or more limbs of the user.

In another embodiment, the memory further contains instructions, thatwhen executed by the processor, cause the processor to perform the stepof determining a facial direction of the user.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a system in accordance with embodiments of thepresent invention.

FIG. 2 is a diagram of a client device in accordance with embodiments ofthe present invention.

FIG. 3 is a block diagram of components in accordance with embodimentsof the present invention.

FIG. 4A-4D show a shock mount in accordance with embodiments of thepresent invention.

FIG. 5 illustrates an example of a shock mount in use.

FIG. 6 illustrates a weapon simulator in accordance with embodiments ofthe present invention.

FIG. 7 illustrates a weapon simulator in accordance with additionalembodiments of the present invention.

FIG. 8A-8 i illustrate examples of inverse kinematic solvercomputations.

FIG. 9A is a front view of a wearable sensor in accordance withembodiments of the present invention.

FIG. 9B is a rear view of the wearable sensor of FIG. 9A.

FIG. 10 is a flowchart indicating process steps for embodiments of thepresent invention.

FIG. 11 is a flowchart indicating additional process steps forembodiments of the present invention.

FIG. 12 is an exemplary user interface in accordance with embodiments ofthe present invention.

FIG. 13 shows an additional embodiment of a weapon simulator.

The structure, operation, and advantages of disclosed embodiments willbecome further apparent upon consideration of the following descriptiontaken in conjunction with the accompanying figures (FIGs.). The figuresare intended to be illustrative, not limiting. Certain elements in someof the figures may be omitted, or illustrated not-to-scale, forillustrative clarity.

DETAILED DESCRIPTION

Disclosed embodiments provide systems and methods for conducting afirearm usage simulation including simulated firearm discharge, enablingenhanced training of armed forces and/or law enforcement personnel. Amotion tracking system tracks motion of one or more users. Inembodiments, the users wear one or more sensors, transponders, or otherwearable devices on their bodies to allow tracking by the motiontracking system. A scenario management system utilizes position,orientation, and motion information provided by the motion trackingsystem to evaluate user performance during a scenario. A weaponsimulator includes sensors that indicate position of the weapon and/ororientation of the weapon. The weapon simulator may further providetrigger activation indications to the scenario management system. Inembodiments, the scenario management system generates, plays, reviews,and/or evaluates simulations. The evaluation can include scoring basedon reaction times, posture, body position, body orientation, and/orother attributes. Thus, disclosed embodiments provide a safe andcost-effective way to train personnel in the use of a weapon in variousscenarios, and evaluate the users based on information provided by themotion tracking system and/or weapon simulator.

FIG. 1 is a diagram of a system 100 in accordance with embodiments ofthe present invention. System 100 includes a scenario management system102. The scenario management system 102 can be implemented in anelectronic computing device that includes a processor 140, a memory 142coupled to the processor, and storage 144, also coupled to the processor140. Memory 142 contains instructions 147, that when executed by theprocessor 140, cause the scenario management system 102 to perform stepsin accordance with embodiments of the present invention. Memory 142 maybe a non-transitory computer readable medium. Memory 142 may include,but is not limited to, flash, read-only memory (ROM), staticrandom-access memory (SRAM), magnetic storage, optical storage, or othersuitable storage mechanism. Storage 144 may include one or more magnetichard disk drives (HDD), solid state disk drives (SSD), optical storagedevices, tape drives, and/or other suitable storage devices.

In embodiments, storage 144 may include multiple hard disk drivesconfigured in a RAID (redundant array of independent disks)configuration. In embodiments, the RAID configuration can include a RAID1 configuration in which data is copied seamlessly and simultaneously,from one disk to another, creating a replica, or mirror. If one harddisk drive becomes inoperable, another hard disk drive continues tooperate, providing a level of fault tolerance.

The processor 140 may include multiple cores. In embodiments, thescenario management system 102 may include multiple processors, whereeach processor includes multiple cores. Embodiments may schedule tasksand threads over multiple processors and/or cores to achieve a level ofparallelism in execution of various tasks such as computations,rendering, and/or scenario generation.

Embodiments may include additional redundancy through failovermechanisms. In embodiments, the scenario management system 102 may beinstantiated as a virtual machine operating in a cloud environment. Inembodiments, multiple instantiations of the scenario management system102 may be implemented in the cloud environment. Scenario managementsystem 102 is in communication with network 124. Network 124 may be theInternet, a local area network (LAN), wide area network (WAN), oranother suitable network.

The term “Internet” as used herein refers to a network of networks whichuses certain protocols, such as the TCP/IP protocol, and possibly otherprotocols such as the hypertext transfer protocol (HTTP) for hypertextmarkup language (HTML) documents that make up the World Wide Web (web).The physical connections of the Internet and the protocols andcommunication procedures of the Internet are well known to those ofskill in the art. Access to the Internet can be provided by Internetservice providers (ISP). Users on client systems, such as client 116obtains access to the Internet through the Internet service providers.Access to the Internet allows users of the client computer systems toexchange information, receive and send e-mails, and view documents, suchas documents which have been prepared in the HTML format. Thesedocuments are often provided by web servers which are considered to be“on” the Internet. Often these web servers are provided by the ISPs,although a computer system can be set up and connected to the Internetwithout that system being also an ISP as is well known in the art.

System 100 further includes motion tracking system 130. Motion trackingsystem 130 includes a processor 132, a memory 134 coupled to theprocessor, and storage 136, also coupled to the processor 132. Memory134 contains instructions, that when executed by the processor 132,cause the motion tracking system 130 to perform steps in accordance withembodiments of the present invention. Memory 134 may be a non-transitorycomputer readable medium. Memory 134 may include, but is not limited to,flash, read-only memory (ROM), static random-access memory (SRAM),magnetic storage, optical storage, or other suitable storage mechanism.Storage 136 may include one or more magnetic hard disk drives (HDD),solid state disk drives (SSD), optical storage devices, tape drives,and/or other suitable storage devices. In embodiments, memory 134includes at least 32 gigabytes of RAM. Motion tracking system 130further includes one or more cameras 137. The cameras may be configuredto detect visible and/or infrared light. The motion tracking system 130may further include one or more sensors. These sensors may include, butare not limited to, temperature sensors, proximity sensors, noisesensors, and/or other suitable sensors. In some embodiments, processor132 may include an Intel i7 CPU or higher. The motion tracking system130 may include a graphics processor such as the Nvidia GTX 1070 orhigher, and include a high-performance network card.

A client device, indicated as 116 may be connected to network 124 via awired or wireless interface. Client device 116 may include a mobilecomputing device such as a smartphone or tablet, a laptop computer, adesktop computer, or other suitable computing device. The client-serverarchitecture allows a user to remotely access features of the scenariomanagement system 102. In embodiments, the client device 116 may includean Intel i7 CPU or higher, an Nvidia GTX 1080 graphics processing unitor higher, and 16 GB of ram or more.

Embodiments of the present invention may utilize a JavaScript ObjectNotation (JSON) web service to make a JSON call to the scenariomanagement system. In some examples, the JSON call is made using XMLHTTP, which implements an XML HTTP object that has functionalityenabling the exchange of Extensible Markup Language (XML) data directlyover the Internet using the Hypertext Transfer Protocol (HTTP). The XMLHTTP object allows access of the scenario management system data from aserver, parsing the data using an XML Document Object Model (DOM), andposting XML data through a standard firewall directly to an HTTP server.

The cameras 137 and/or sensors 138 of motion tracking system 130 may bedeployed in a venue such as a room, building, or outdoor area, such thatthey can track the motion of one or more users (200A, 200B, 200C, and200D). Note that while four users are shown in FIG. 1, in practice, moreor fewer users may be present. Referring now to user 200A, additionaldetail is shown. Each user may utilize one or more wearable sensors 206.The wearable sensors can be used to detect motion and position of auser. By having sensors on the limbs of a user, the position and/ororientation of a user can be more precisely determined. Each user mayfurther wear a helmet 205. The helmet 205 may include a sensor 207 thatcan be used to determine head location and/or head orientation of thewearer (user). Each user may further wear goggles 204. The goggles 204may include virtual reality goggles, augmented reality goggles, or othersuitable eyewear. The goggles 204 may further include speakers thatprovide audible feedback to a user. In embodiments, the goggles 204 mayinclude a forward-facing camera to enable recording and/or monitoring ofthe user's point of view. In embodiments, the goggles 204 may include auser-facing camera to monitor the eye gaze of a user. In this way, thedirection of eye gaze, even if different from head orientation, can beascertained.

Each user may further utilize a weapon simulator 202. The weaponsimulator may be in the form of a firearm, and may include a recoilsimulation mechanism such as compressed air or a mechanical spring tosimulate the recoil associated with discharging a real firearm. Theweapon simulator 202 may further include an inertial tracker 208 affixedto the weapon simulator 202. The inertial tracker 208 may include one ormore accelerometers and/or gyroscopes to track motion of the weaponsimulator 202. Sometimes, motion associated with recoil can adverselyaffect the inertial tracker 208. Therefore, embodiments may furtherinclude a shock mount 210 disposed between the weapon simulator 202 andthe inertial tracker 208. This allows the position of the weaponsimulator to be tracked as the user moves it, while preventing theadverse effects of recoil motion, since the shock mount absorbs some ofthe recoil motion.

FIG. 2 is a block diagram of a client device 300 in accordance withembodiments of the present invention. In embodiments, client device 300is an electronic device that may include a desktop computer, laptopcomputer, tablet computer, smartphone, and/or other suitable clientdevice. Client device 300 may be similar to client device 116 as shownin FIG. 1. Client device 300 includes a processor 302, a memory 304coupled to the processer 302, and storage 306. The memory 304 may be anon-transitory computer readable medium. Memory 304 may include RAM,ROM, flash, EEPROM, or other suitable storage technology. The memory 304contains instructions, that when executed by processor 302, enablecommunication to/from scenario management system 102 of FIG. 1. Clientdevice 300 further includes a network communication interface 310 forperforming this communication. In embodiments, network communicationinterface 310 includes a wireless communications interface such as acellular data interface and/or a Wi-Fi interface. In embodiments, thestorage 306 includes flash, SRAM, one or more hard disk drives (HDDs)and/or solid state disk drives (SDDs).

Device 300 may further include a user interface 308. User interface 308may include a keyboard, monitor, mouse, and/or touchscreen, and providesa user with the ability to enter information as necessary to utilizeembodiments of the present invention. In embodiments, a user uses thedevice 300 to access the scenario management system 102.

FIG. 3 is a block diagram 400 of components in accordance withembodiments of the present invention. Components include, but are notlimited to, motion tracker 402, weapons interface 404, inverse kinematicsolver 406, client communication component 408, scenario managementcomponent 410, and/or data analysis 412. The motion tracker 402 may be acommercially available motion tracker such as OptiTrack™ byNatrualPoint, of Corvallis, Oreg. The motion tracker 402 may includemultiple cameras installed within a room, building, or other venue wherea scenario is to be executed. The cameras may be adapted to trackwearable sensors that emit infrared light. As a user, wearing thewearable sensors moves, his/her motion is tracked b the motion tracker402. The weapons interface 404 may implement recoil simulation, gunflash simulation, trigger activation notifications, and/or otherweapons-related simulation functions. The inverse kinematic (IK) solver406 may be used to create an inverse kinematic solution to rotate andposition links in a chain. This can be used to model human movement tocreate virtual targets and/or model the motion of live users.

The scenario management component 410 may be implemented as a computersystem such as system 102 of FIG. 1. Alternatively, the scenariomanagement component 410 may be implemented as multiple computers in adistributed or cloud computing environment. The scenario managementcomponent may be used for generating scenarios, executing scenarios,and/or playback of previously executed scenarios. The data analysiscomponent 412 may be used for analyzing an executed scenario. This caninclude performing numerous assessments on users that are tracked by themotion tracker 402. Assessments can include reaction time, locationanalysis, physiological orientation of the user, orientation of theuser's head, eye gaze, limb position, and others.

For example, in a given scenario, upon hearing a gunshot, a user may betrained to drop to a crouched position, turn his head towards thedirection of the gunshots, and draw his weapon. To provide a trainingexperience for a user, a scenario is executed by the scenario managementcomponent 410. This component may generate the virtual environment,including audiovisual information rendered by goggles 204. A gunshotsound is rendered on the speaker of goggles 204 worn by the user, andthe scenario management component 410 records this time. The user startsto move, and the motion tracker determines how long it takes to get intoa crouched position, how long it takes for the user to draw his weapon,and if his head is oriented in the proper position. In embodiments,after a predetermined time (e.g. 2 seconds), the user orientation andweapon orientation are evaluated. In embodiments, a user score isgenerated based on the time required for the user to achieve the properposition. In this way, the user can continually practice, and review hisperformance to achieve an optimal reaction time. Other, more complexexamples are possible, such as scenarios utilizing multiple friends andfoes. In embodiments, one or more of the friends and/or foes may bevirtual.

FIG. 4A-4D show a shock mount in accordance with embodiments of thepresent invention. FIG. 4A is a perspective view of a shock mount 500 inaccordance with embodiments of the present invention. Shock mount 500includes a top plate 503, a bottom plate 506, and a plurality ofresilient members 504 configured and disposed between the top plate 503and the bottom plate 506, and terminated at the top with end caps,indicated generally as 541. This reduces the amount of shock imparted tothe inertial tracker 508 from the handle 526 of a weapon simulator as aresult of a recoil simulation. The weapon simulator may utilizecompressed air, solenoids, springs, or other electromechanicalmechanisms to simulate a recoil that occurs when a firearm isdischarged. The shock mount 500 reduces the shock imparted on theinertial tracker 508, which helps improve the reliability of datareturned by the inertial tracker. In embodiments, the inertial trackermay be a commercially available inertial tracker such as Vive™ trackerby HTC corporation. In some embodiments, the resiliency of the resilientmembers 504 may be adjustable to accommodate different types of weaponsimulators and/or inertial trackers. Shock mount 500 may be similar toshock mount 210 shown in FIG. 1. FIG. 4B is a top view of the shockmount 500, illustrating the mounting shoe 543 configured and disposed toreceive a weapon simulator. FIG. 4C is a front view of the shock mount500, illustrating a plurality of coupling pins 545 for interfacing to aninertial tracker. In embodiments, the coupling pins 545 may be springloaded “Pogo” pins. FIG. 4D is a bottom view of the shock mount 500,indicating a linear arrangement of the coupling pins 545.

FIG. 5 illustrates an example of a shock mount in use. A weaponsimulator 520 has a handle 526. In embodiments, the weapon simulator 520may be in the form factor of a pistol. The shock mount 500 is affixed tothe bottom of the handle (grip) 526. The inertial tracker 508 is affixedto the shock mount 500. The weapon simulator 520 shown is a Glocksimulator. In embodiments, the weapon simulator 520 is configured to usecompressed gas in order to create haptic feedback (or recoil) thatcreates a strong vibration to the inertial tracker 508 tracker. Theshock mount 500 of disclosed embodiments connects the inertial tracker508 to the handle 526 while also reducing the shock from simulatedrecoil which could possibly ruin the tracking data of the inertialtracker 508. In embodiments, a small pressure-sensitive button is placedbehind the trigger to send the trigger activation information throughelectronic circuitry that connects to an interface of the inertialtracker. The inertial tracker may then use a wireless interface such asBluetooth™ to send the trigger activation signal to the computer forfurther processing.

FIG. 6 illustrates a weapon simulator 600 in accordance with embodimentsof the present invention. Weapon simulator 600 is in the form factor ofan automatic rifle. A motor 602 embedded in the grip is configured togenerate haptic feedback upon activation of the trigger 604. A battery612 in the stock of the weapon simulator 600 provides power for themotor 602. Additionally, a wireless transmitter 606 is configured anddisposed to provide discharge information to the scenario managementsystem. The wireless transmitter 606 may communicate with a wirelessreceiver 608 attached to a computer 610. In embodiments, computer 610 ispart of the scenario management system. Thus, embodiments include acommunication module configured and disposed to wirelessly transmit atrigger indication to the scenario management system in response toactivation of the trigger.

FIG. 7 illustrates a weapon simulator 700 in accordance with additionalembodiments of the present invention. Weapon simulator may be in theform factor of a pistol. An optical tracker 702 may be installed on theweapon simulator 700. One or more light sources, indicated as 704 and706 may be tracked by the motion tracking system to provide informationon the position and/or orientation of the weapon simulator to thescenario management system.

In embodiments, the tracker 702 orients one or more LEDs in specificlocations for optimal tracking. A pressure sensitive button is placedbehind the handgun trigger and that passes a signal to a computer via awireless communication protocol such as Bluetooth™, WiFi, Zigbee, orother suitable protocol upon activation of the trigger 708 of the weaponsimulator. In embodiments, the unit is self-contained with its own LIPObattery, voltage converter, charger port, and on/off button.

FIG. 8A-8 i illustrate examples of inverse kinematic solvercomputations. In embodiments, a full body IK solver allows animation ofa human body model in real time based on animations of 5 control points.In embodiments, a body skeleton is divided into the following five bonechains. A spine chain starts with a pelvis bone and ends with a headbone. A left leg chain starts with a pelvis bone and ends with a leftfoot bone. A right leg chain starts with a pelvis bone and ends with aright foot bone. A left arm chain starts with a left shoulder bone andends with a left palm bone. A right arm chain starts with a rightshoulder bone and ends with a right palm bone. Each chain of bones issolved using a Backward/Forward IK solver gets as an input, two 3Dmatrices and calculates transform of bones in a chain in 3D space.Changing locations or rotations of either start or end input will changetransformation of the bones in the chain. Each chain can consist of anynumber of bones. Referring now to FIG. 8A, a bone chain 800 is shown inan extended bone state. Three bones (802, 803, and 804) are extendedfully such that endpoint 801 and endpoint 805 are at a maximum possibledistance.

Referring now to FIGS. 8B, 8C, and 8D, there are shown examples ofnon-extended bone states. Referring to FIG. 8B, bone chain 810 isoriented such that the distance D between first endpoint (start) 811 andsecond endpoint (end) 812 is less than the sum of the lengths of allbones in the bone chain. A similar case exists for bone chain 814 ofFIG. 8C and bone chain 815 of FIG. 8D.

Referring now to FIG. 8E there is a bone chain with a force vector UPapplied to it. In order to bend the chain in the known direction, the UPdirection has to be defined for the chain. UP direction is a 3D vector.UP direction gets applied to the initial pose and changes it. In thediagram, the UP direction is to the left, which means the chain 819 willbend to the right.

Referring now to FIG. 8F, forward solving computations are shown.Starting from the current state of the chain, positions of each bone inthe chain are found using the forward solving equation:

A′=End′−(End′−A)

Starting from the bone closest to the end, this equation is repeated foreach bone by replacing End′ vector with the location of previous bone inthe chain.

For finding B′ for example, the following equation is used:

B′=A′−(A′B)

Similarly, for C′:

C′=B′−C)

Referring now to FIG. 8G. a chain rigidity factor is applied. Inembodiments, each bone has a rigidity factor which determines how easilya given bone will change its position when start or end transformschange. A value from 0 to 1 determines where the bone will be placedbetween the initial pose and the current pose. In this example, bone 821has a rigidity factor of 0.5 and so it is moved to halfway betweenpoints B and B′.

Referring now to FIG. 8H, backward solving is performed. Starting fromthe current state of the chain, positions of each bone in the chain arefound using a backward solving equation. The process starts from thebone closest to the start input, and the equations below are applied:

C″=C

B″=C″+(B′−C″)

A″=B″+(A′−B″)

Referring now to FIG. 8i , the orientation of the bones is calculated.Bone chain 823 shows an initial position, and bone chain 825 shows afinal position. In embodiments, orientation of the bones gets calculatedafter positions are solved for, and it is known where each bone islocated and where this bone was in the initial pose. In embodiments, inthe initial pose a rotational matrix is constructed for each bone. Inthe solved pose there is constructed a new rotational matrix for eachbone using the same axis order. One axis is parallel to the length ofthe bone (Z in the diagram). A second axis is linearly interpolatedbetween the upnode (UP) chain vector multiplied by start and upnodechain vector multiplied by end transform (Y in the diagram). Linearinterpolation of this axis allows simulation of twisting of bones alongtheir long axis (z) if end or start transform is twisted. This approachallows adjusting of the extent of the twist for each bone easily bychanging one numeric value. Once there are derived two rotationalmatrices for the bone, then the offset between them is calculated andthis transform offset is applied to the bone. This approach allowsrotation of bones independently of their initial orientation.Orientation of bones can be inconsistent in the initial chain pose.

FIG. 9A is a front view of a wearable sensor 900 in accordance withembodiments of the present invention. Wearable sensor 900 includes lightemitting diode 902 installed therein. A battery within the sensor (notshown) applied power to the light emitting diode 902. In embodiments,the light emitting diode (LED) 902 emits infrared light and does notemit substantial visible light. This allows the LED 902 of sensor 900 tobe detected by the motion tracking system, and yet not be noticeable tothe users.

FIG. 9B is a rear view of the wearable sensor 900 of FIG. 9A. In thisview, the adjustable strap 904 can be seen. This strap 904 is used tosecure the wearable sensor around a limb (arm, leg) of a user. Inembodiments, the strap 904 may have a fastener such as a snap, hook andloop fastener, or other suitable fastening mechanism. Wearable sensor900 may be similar to sensors 206 shown in FIG. 1.

FIG. 10 is a flowchart 1000 indicating process steps for embodiments ofthe present invention. At process step 1002, a user position isdetermined. The user may be wearing one or more wearable devices such asposition sensors and/or position indicators. The wearable devices mayemit infrared light that is tracked by a motion tracking system. Inprocess step 1004, an initial physiological orientation is determined.In some embodiments, a wearable device is worn on each limb, allowingthe motion tracking system to determine physiological orientation. Thephysiological orientation can include a stance (e.g. standing,crouching, prone), and may further including a direction the user isfacing. In process step 1006, a simulated firearm discharge is detected.In embodiments, this can include a weapon simulator in the form factorof a firearm. The weapon simulator includes a power source (battery) anda wireless transmitter that is configured and disposed to transmit atrigger activation indication to a computer to indicate that the weaponsimulator has discharged (e.g. a user pulled the trigger on the weaponsimulator).

The weapon simulator may or may not actually shoot a projectile. Inprocess step 1008, a user reaction is determined. This can includedetermining a second physiological orientation of the user at somepredetermined time after the detection of the simulated firearmdischarge. The second physiological orientation can include a positionof a person, position of the limbs of that person, orientation of thetorso of the person (which way the torso is facing), and/or orientationof the head of the person (which way the head of the user is facing).Thus, in embodiments, determining a physiological orientation of theuser comprises determining a facial direction of the user.

Additionally, in embodiments with a user-facing camera (e.g. included invirtual or augmented reality goggles), an eye gaze direction may furtherbe included in the physiological orientation. Various attributes canthen be evaluated, including, but not limited to, the time required forthe user to achieve the second physiological orientation (e.g. a measureof how long did it take the user to get into a crouch position inresponse to hearing and/or seeing a simulated firearm discharge), thecorrectness of the second physiological orientation as compared to aknown orientation (e.g. a measure of if the user's crouch issufficiently low), and/or the time required to draw a weapon (e.g. ameasure of the time required for the user to pull a weapon simulatorfrom a holster and orient it in a position to fire). Other attributes ofuser reaction may be evaluated instead of, or in addition to theaforementioned attributes in embodiments of the present invention.

FIG. 11 is a flowchart 1100 indicating additional process steps forembodiments of the present invention. In process step 1102, a finalposition time is determined. This may include determining a time wherethe user's level of motion is below a predetermined threshold. Thus,embodiments include further comprising measuring a duration from a timeof the simulated firearm discharge to a time of detecting the determinedphysiological orientation. As an example, when a user is in the processof quickly moving from a standing position to a crouched position, thereis a relatively large amount of motion. Once the user is stabilized inthe crouched position, there is relatively less motion. The time whenthe user is substantially still may be recorded as a final positiontime. At process step 1104, a final position quality is determined. Thismay include utilization of motion tracking data to determine how closethe user's physiological orientation is to an expected physiologicalorientation. This can include an assessment of if the user is in aproper stance/position, facing the proper direction, appropriatelydrawing his weapon simulator, and/or other factors. In process step1106, the final position time is compared to average times. In processstep 1108, a score is generated based on the comparison to the averagetime. In some embodiments, the score may be generated using the formula:

K(1−X)

Where:

K is a constant;X is the final position time in seconds.

As an example, if an average time for a law enforcement professional togo from standing to crouched is 400 milliseconds, then the finalposition time may be compared against the average time using the aboveformula. If K is 116, then the following score is achieved for differentcrouch times:

Crouch time (milliseconds) Score 400 69.6 500 50 600 46.4 200 92.8 30081.2

As can be seen from the table above, the formula returns a score suchthat a user with an average time of 400 milliseconds earns a score ofabout 70, while a user with a score of 300 returns a higher score(81.2). Similarly, a user with a relatively slow time of 600milliseconds returns a low score of 46.4. This concept can also beapplied to other attributes, such as position quality, weapon draw time,weapon discharge time, and/or other attributes. In embodiments, thescore for each attribute may be combined into an overall score.

FIG. 12 is an exemplary user interface 1200 in accordance withembodiments of the present invention. User interface 1200 may beimplemented on a client device such as client device 116 of FIG. 1. Theuser interface may be used to allow users and/or trainers and coaches toreview performance. User interface 1200 includes a video window 1202that may show video and/or a virtual animated rendition of the userperformance. In some embodiments, real video and animated graphics maybe composited together in video window 1202. User interface 1200 furtherincludes a name field 1204, a scenario name 1206, a final position score1208, a reaction time 1210, a weapon draw time 1214, and a weapondischarge time 1216. In embodiments, the weapon draw time 1216 may bethe time required by the user to move his weapon simulator from aholstered position to a ready position as determined by the motiontracking system. In embodiments, the weapon discharge time may be thetime required by the user to fire the weapon simulator after getting theweapon simulator into the ready position (e.g. aimed at a target). Thedata analysis component (412 of FIG. 3) may further providecomputer-generated recommendations, which may be rendered in field 1218.The recommendations can be based on scores and/or times for one or moreattributes during a scenario execution. The user interface 1200 ismerely exemplary, and other user interfaces are possible showing more,fewer, or different fields in some embodiments.

FIG. 13 shows an additional embodiment. Weapon simulator 1300 includesadditional stock 1302 affixed to the rear end of the weapon simulator.This enables users wearing a head-mounted display (HMD) to get into ashooting position without having the HMD bump into the stock end of theweapon simulator, thereby allowing for a more unencumbered userexperience.

Some embodiments may provide an untethered experience with computersmounted in backpacks. Such embodiments may include, but are not limitedto, an MSR V1 with a GTX 1070, or an HP ZVR with a GTX 1070. Thecomputers may be configured with an Intel i7 processor and at least 16GB of RAM. Other backpack computers are possible in embodiments of thepresent invention.

As can now be appreciated, disclosed embodiments provide an improvedtraining system for firearm usage. A motion tracking system tracks themotion of one or more users. Users use a weapon simulator thatintegrates with a scenario management system. The scenario managementsystem thus obtains information about weapon position and weapondischarge, as well as position of the users. The scenario managementsystem can generate scenarios where live participants, virtualparticipants, and/or computer-generated targets work together or againsteach other to conduct training drills. Through the use of virtualreality and/or augmented reality, various landscapes, terrain,buildings, and other factors can be simulated. The reactions of userscan be timed and assessed, allowing for improved review of theperformance of users such as military and law enforcement personnel. Inthis way, the effectiveness and safety of these people can becontinuously monitored and improved.

The present invention may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a non-transitory computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

Each of the above methods may be executed on one or more processors onone or more computer systems. Embodiments may include various forms ofdistributed computing, client/server computing, and cloud basedcomputing. Further, it will be understood that the depicted steps orboxes contained in the disclosed flowcharts are solely illustrative andexplanatory. The steps may be modified, omitted, repeated, or re-orderedwithout departing from the scope of this disclosure. Further, each stepmay contain one or more sub-steps. While the foregoing drawings anddescription set forth functional aspects of the disclosed systems, noparticular implementation or arrangement of software and/or hardwareshould be inferred from these descriptions unless explicitly stated orotherwise clear from the context. All such arrangements of softwareand/or hardware are intended to fall within the scope of thisdisclosure.

The block diagrams and flowchart illustrations depict methods,apparatus, systems, and computer program products. Any and all suchfunctions, generally referred to herein as a “circuit,” “module,” or“system” may be implemented by computer program instructions, byspecial-purpose hardware-based computer systems, by combinations ofspecial purpose hardware and computer instructions, by combinations ofgeneral purpose hardware and computer instructions, and so on.

It will be understood that a computer may include a computer programproduct from a computer-readable storage medium and that this medium maybe internal or external, removable and replaceable, or fixed. Inaddition, a computer may include a Basic Input/Output System (BIOS),firmware, an operating system, a database, or the like that may include,interface with, or support the software and hardware described herein.

Embodiments of the present invention are neither limited to conventionalcomputer applications nor the programmable apparatus that run them. Toillustrate: the embodiments of the presently claimed invention couldinclude an optical computer, quantum computer, analog computer, or thelike. A computer program may be loaded onto a computer to produce aparticular machine that may perform any and all of the depictedfunctions. This particular machine provides a means for carrying out anyand all of the depicted functions.

Any combination of one or more computer readable media may be utilizedincluding but not limited to: a non-transitory computer readable mediumfor storage; an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor computer readable storage medium or anysuitable combination of the foregoing; a portable computer diskette; ahard disk; a random access memory (RAM); a read-only memory (ROM), anerasable programmable read-only memory (EPROM, Flash, MRAM, FeRAM, orphase change memory); an optical fiber; a portable compact disc; anoptical storage device; a magnetic storage device; or any suitablecombination of the foregoing. In the context of this document, acomputer readable storage medium may be any tangible medium that cancontain or store a program for use by or in connection with aninstruction execution system, apparatus, or device.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device. Program data may also bereceived via the network adapter or network interface.

These computer readable program instructions may be provided to aprocessor of a computer, or other programmable data processing apparatusto produce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks. These computerreadable program instructions may also be stored in a computer readablestorage medium that can direct a computer, a programmable dataprocessing apparatus, and/or other devices to function in a particularmanner, such that the computer readable storage medium havinginstructions stored therein comprises an article of manufactureincluding instructions which implement aspects of the function/actspecified in the flowchart and/or block diagram block or blocks.

It will be appreciated that computer program instructions may includecomputer executable code. A variety of languages for expressing computerprogram instructions may include without limitation C, C++, Java,JavaScript™, assembly language, Perl, Python, Ruby, hardware descriptionlanguages, database programming languages, functional programminglanguages, imperative programming languages, and so on. In embodiments,computer program instructions may be stored, compiled, or interpreted torun on a computer, a programmable data processing apparatus, aheterogeneous combination of processors or processor architectures, andso on. Without limitation, embodiments of the present invention may takethe form of web-based computer software, which includes client/serversoftware, software-as-a-service, peer-to-peer software, or the like.

In embodiments, a computer may enable execution of computer programinstructions including multiple programs or threads. The multipleprograms or threads may be processed approximately simultaneously toenhance utilization of the processor and to facilitate substantiallysimultaneous functions. By way of implementation, any and all methods,program codes, program instructions, and the like described herein maybe implemented in one or more threads which may in turn spawn otherthreads, which may themselves have priorities associated with them. Insome embodiments, a computer may process these threads based on priorityor other order.

Unless explicitly stated or otherwise clear from the context, the verbs“execute” and “process” may be used interchangeably to indicate execute,process, interpret, compile, assemble, link, load, or a combination ofthe foregoing. Therefore, embodiments that execute or process computerprogram instructions, computer-executable code, or the like may act uponthe instructions or code in any and all of the ways described.Furthermore, the method steps shown are intended to include any suitablemethod of causing one or more parties or entities to perform the steps.

The terminology used herein is for describing particular aspects onlyand is not intended to be limiting of the invention. As used herein, thesingular forms “a”, “an” and “the” are intended to include the pluralforms as well, unless the context clearly indicates otherwise. It willbe further understood that the terms “include” and “including” when usedin this specification, specify the presence of stated features,integers, steps, operations, elements, and/or components, but do notpreclude the presence or addition of one or more other features,integers, steps, operations, elements, components, and/or groupsthereof. Certain examples and elements described in the presentspecification, including in the claims and as illustrated in thefigures, may be distinguished or otherwise identified from others byunique adjectives (e.g. a “first” element distinguished from another“second” or “third” of a plurality of elements, a “primary”distinguished from a “secondary” one or “another” item, etc.) Suchidentifying adjectives are generally used to reduce confusion oruncertainty, and are not to be construed to limit the claims to anyspecific illustrated element or embodiment, or to imply any precedence,ordering or ranking of any claim elements, limitations or process steps.

While the invention has been disclosed in connection with preferredembodiments shown and described in detail, various modifications andimprovements thereon will become apparent to those skilled in the art.Accordingly, the forgoing examples should not limit the spirit and scopeof the present invention; rather it should be understood in the broadestsense allowable by law.

What is claimed is:
 1. A computer-implemented method executed by asystem for conducting a firearm usage simulation, comprising:determining, by a motion tracker component, an initial position of auser, wherein the motion tracker component comprises one or more camerasdeployed in an area of the firearm usage simulation, the one or morecameras being configured to track infrared light emitted from one ormore wearable sensors worn by the user; determining, by the motiontracker component, the one or more wearable sensors, and a user-facingcamera included within goggles worn by the user, an initialphysiological orientation of the user, wherein a subset of the one ormore wearable sensors are affixed on a helmet worn by the user and areaffixed to one or more limbs of the user; detecting, by a weaponsinterface component, a simulated firearm discharge; and determining, bya data analysis component, a reaction of the user in response to thesimulated firearm discharge.
 2. The method of claim 1, whereindetermining the initial physiological orientation of the user by themotion tracker component and the one or more wearable sensors comprisesdetermining a position of the one or more limbs of the user.
 3. Themethod of claim 1, wherein determining the initial physiologicalorientation of the user by the motion tracker component and the one ormore wearable sensors comprises determining a facial direction of theuser.
 4. The method of claim 1, wherein determining the initialphysiological orientation of the user comprises determining an eye gazedirection of the user by the user-facing camera included within thegoggles.
 5. The method of claim 1, further comprising: comparing, by thedata analysis component, a second physiological orientation of the userto an expected physiological orientation of the user; and generating, bythe data analysis component, a user score based on the comparison. 6.The method of claim 5, wherein comparing the second physiologicalorientation of the user to the expected physiological orientation of theuser, by the data analysis component, includes comparing a position ofthe one or more limbs of the user.
 7. The method of claim 5, whereincomparing the second physiological orientation of the user to theexpected physiological orientation of the user, by the data analysiscomponent, includes comparing a facial direction of the user to anexpected facial direction.
 8. The method of claim 1, wherein the systemis a motion capture virtual reality system comprising the motion trackercomponent, the weapons interface component, and the data analysiscomponent, and wherein the one or more wearable sensors are activesensors.
 9. A weapon usage evaluation system, comprising: a motiontracking system comprising one or more cameras deployed in an area of afirearm usage simulation, configured and disposed to track infraredlight emitted from one or more wearable sensors worn by one or moreusers; a scenario management system, configured and disposed to receivemotion information of the one or more users from the motion trackingsystem; and a weapon simulator configured and disposed to providedischarge information associated with the firearm usage simulation andposition information of the one or more users to the scenario managementsystem including at least reaction time information.
 10. The system ofclaim 9, further comprising an inertial tracking device affixed to theweapon simulator.
 11. The system of claim 10, further comprising a shockmount configured and disposed between the weapon simulator and theinertial tracking device.
 12. The system of claim 11, wherein the shockmount comprises: a top plate; a bottom plate; and a plurality ofresilient members configured and disposed between the top plate and thebottom plate.
 13. The system of claim 9, wherein the weapon simulatorcomprises: a trigger; a motor configured and disposed to generate avibration in response to activation of the trigger; and a communicationmodule configured and disposed to wirelessly transmit a triggerindication to the scenario management system in response to activationof the trigger.
 14. The system of claim 9, further comprising an inversekinematic solver.
 15. The system of claim 9, further comprising: aprocessor; a memory coupled to the processor, wherein the memorycontains instructions, that when executed by the processor, cause theprocessor to perform steps of: determining an initial position of auser; determining an initial physiological orientation of the user;detecting a simulated firearm discharge; and determining a reaction ofthe user in response to the simulated firearm discharge.
 16. The systemof claim 15, wherein the memory further contains instructions, that whenexecuted by the processor, cause the processor to perform the step ofdetermining a position of one or more limbs of the user.
 17. The systemof claim 15, wherein the memory further contains instructions, that whenexecuted by the processor, cause the processor to perform the step ofdetermining a facial direction of the user.
 18. The system of claim 9,wherein the weapon usage evaluation system is a motion capture virtualreality weapon usage evaluation system, and wherein the one or morewearable sensors are active sensors.